-
Notifications
You must be signed in to change notification settings - Fork 3
Add --blacklist/--whitelist to mxqsub, add mxset #82
Conversation
Detected some unbalance between "disabled-servers" and "whitelisted-servers" blacklisted? enabled? hosts .vs. servers? Also it became quit long:
Suggestions? |
|
+=1 |
Well, but currently we have "--whitelisted-servers=" and "--update-whitelisted-servers" for mxqset with different meanings. The first option replaces the list, while the second option adds or removes from the list.
I think the feature to add or remove hosts is useful. But there also needs to be a way to clear the list. How could this be done with one option only?
Would this remove all other whitelisted servers from the group?
for whitelist/blacklist or to add/remove from one of the lists? Perhaps use "+" prefix (additional to "-" prefix for deltas) ? What about
But what would be the result of "+host1 +host2 host3 host4"` ? Would the addition of host1 and host2 be reset by the plain "host3" . And if "host3" removed host1 and host2, why didn't "host4" remove "host3" ? Maybe prefix the whole string with "+" or "-" for deltas?
Maybe this would be even more convenient if you put an expression |
oh - wow - this gets philosophical - I would say:
|
Okay, I use that with a slight change: '+' and '-' (update) have same priority and are processed left to right. Doesn't make a difference with your example, but "-x +x a" would result in "a x" instead of "a". |
I don't like |
Ok - good - just something what cam to my mind:
would clear the whitelist. Does this mean that this group will not run until I put something onto the white list, or does a missing whitelist mean "all"? |
mmh - I don't like this white/black philosophy |
Oh, gosh, I need to implement this with special code Yes, an existing whitelist means only the servers on the whitelist, a misisng whitelist means all servers. We might change that to "all normal server" later, so that we can put "special" servers into the cluster (e.g. for development or unstable hardware) which only take jobs, when explicitly requested by whitelisted. |
Is this a joke or serious? I hope this remark has nothing to do with skin color? |
I also thought that I would overreact a bit here. But at least the word "blacklist" was used the first time around 1639 in the UK wiki. Around the same time the British slave trade started. wiki. So I think there is a connection. But you may use these words - now that I have in writing that I raised this point. |
Avoid unneeded warning when scanning JOB_TMPDIR_MNTDIR for possible leftover job mounts. The parent directory of JOB_TMPDIR_MNTDIR is created when the first job with the --tmpdir feature is started. So it won't exist in a new server. This cleanup path is only used in the exceptional case that the usual cleanup path (via job_has_finished) didn't succeed, e.g. when mxqd was killed. The cleanup is not essential for the currently running server.
Add function attribute ((unusged)) to avoid a compiler warning when the static inline function defined in the header file is not used by the compilation unit.
Add header file for the quasi-standard [1] xmalloc call. The current approach implemented in mx_util is to wait and retry on ENOMEM malloc failure. However, this overhead doesn't seem to be justified, because it is difficult to imagine a case, where a malloc would fail with ENOMEM at one time and a retry would succeed. [1] https://www.gnu.org/software/libc/manual/html_node/Malloc-Examples.html
Add a utility module which can store a set of keywords. The set can be created and updated from a string and can be serialized into a string. The canonical string representation produced by this utilits is the space separated list of the sorted keywords in lexical order. Input strings used to create or update a keyword set contain keywords separated by whitespace. The keywords may be prefxied with "+" and "-". For updates, first all unprefixed keywords are processes to create the initial set, than updates are applied from the prefixed keywords. An empty string used for updates is considered "no change", not "set to empty set". Usage example: struct keywordset *kws = keywordset_new("xx yy") keywordset_update(kws, "-yy +zz") // remove yy, add zz keywordset_update(kws, "aa bb") // set to aa, bb if (keywordset_ismember(kws,"bb")) ... // true char *s = keywordset_get(kws): // s now "aa bb" free(s); // caller must free() keywordset_purge(kws) // set nowto empty. keywordset_free(kws);
Add list of blacklisted and whitelisted servers to group. The string is supposed to be canonical (lexical sorted, space separated) as produced by keywordset_get(), so that groups with equal sets can be found by sql string compare.
Add list of blacklisted and whitelisted servers to group. The string is supposed to be canonical (lexical sorted, space separated) as produced by keywordset_get(), so that groups with equal sets can be found by sql string compare.
Add a new options so that specific mxqd servers can be excluded from starting jobs for this group. mxqsub --whitelist "acedia avaritia" sleep 10 // start only on these mxqsub --blacklist dontpanic sleep 10 // start on any node but this one Blacklist has priority, so mxqsub --white "kronos, uselessbox" --black kronos would not start on kronos. The lists of blacklisted and whitelisted servers can be modified with mxqset.
Add a new command which can be used to modify an existing group. Examples: mxqset group 123 --closed mxqset group 123 --open mxqset group 123 --blacklist "dontpanic" # replace mxqset group 123 --whitelist "+uselessbox +gula" # add mxqset group 123 --blacklist -dontpanic # remove mxqset group 123 --whitelist "" # clear The flags open and closed can be set from mxqadmin as well (`mxqadmin --close=123`) , but the synopsis, in which the options take the groupid as a value is difficult to expand to new options, which take a value.
Check, whether this server is qualified to start jobs from a group. Lazy-evaluate qualification criteria defined in the group and cache the result. The qualification criteria need to be reevaluated when the active groups were reloaded, because they may be changed by the user in existing groups. For now, the only qualification criteria are the groups blacklist and whitelist. If a group has a whitelist, the short or long hostname of the mxqd server needs to be on that list, otherwise the server is not qualified for the group. If the servers name is on the blacklist of the group, the server is not qualified for the group. Don't start jobs we are not qualified for. This can later be expanded to additional criteria (e.g. hostconfig or processor flags).
40b5dd3
to
933bd39
Compare
closed for #83 |
Add "whitelisted-servers" on top of "disabled servers"